Context

Previously, Filebeat and Logstash (our utilities for logcollection to OpenSearch) were only deployed through the debugd, which only got deployed in debug-clusters. We can also deploy Filebeat and Logstash through Helm as a Kubernetes deployment, allowing us to also deploy it in non-debug clusters. This should also be helpful in resolving this bug, as we are able to retrieve systemd-logs after the debugd runs. (i.e. when Kubernetes runs)

Proposed change(s)

• Create a hack-script hack/logcollector which retrieves and templates the configuration files into the format that the Elastic Helm charts need. This seems to be very complex for a small problem, but it will allow us to keep a consistent log format between non-debug and debug cluster logs, as we only specify the output format in debugd/{filebeat,logstash} and don't have duplication.
• Create a workflow which deploys Filebeat and Logstash in e2e runs. (temporarily deploy without any condition, i.e. also in debug clusters, to resolve the above mentioned bug)

TODO

• Write documentation on how to deploy in non-CI clusters
• Use a "real" CI condition for the deployment rather than deploying to all clusters TODO added in code, for now deploy both in all runs to resolve "missing OpenSearch logs"-bug - during this time, this will lead to duplicated logs in non-buggy debug clusters.
• Potentially do modifications to filebeat config if https://discuss.elastic.co/t/filebeat-k8s-deployment-duplicated-filestream-id/338008 turns out not to be a bug
• Test CI container build of debugd-{filebeat,logstash} Workflow run
• Possibly extend metadata-fields as viable (e.g. csp)

Checklist

• Update docs
• Add labels (e.g., for changelog category)
• Is PR title adequate for changelog?
• Link to Milestone